Predicting the Location of "interactees" in Novel Human-Object Interactions

Understanding images with people often entails understanding their interactions with other objects or people. As such, given a novel image, a vision system ought to infer which other objects/people play an important role in a given person’s activity. However, while recent work learns about action-specific interactions (e.g., how the pose of a tennis player relates to the position of his racquet when serving the ball) for improved recognition, they are not equipped to reason about novel interactions that contain actions or objects not observed in the training data. We propose an approach to predict the localization parameters for “interactee” objects in novel images. Having learned the generic, actionindependent connections between (1) a person’s pose, gaze, and scene cues and (2) the interactee object’s position and scale, our method estimates a probability distribution over likely places for an interactee in novel images. The result is a human interaction-informed saliency metric, which we show is valuable for both improved object detection and image retargeting applications.